Elastic Information Bottleneck

Information bottleneck is an information-theoretic principle of representation learning that aims to learn a maximally compressed preserves as much information about labels possible. Under this principle, two different methods have been proposed, i.e., (IB) and deterministic (DIB), gained significant progress in explaining the mechanisms deep algorithms. However, these theoretical empirical successes are only valid with assumption training test data drawn from same distribution, which clearly not satisfied many real-world applications. In paper, we study their generalization abilities within transfer scenario, where target error could be decomposed into three components, source error, gap (SG), discrepancy (RD). Comparing IB DIB on terms, prove DIB’s SG bound tighter than IB’s while RD larger IB’s. Therefore, it difficult tell one better. To balance trade-off between RD, propose elastic (EIB) interpolate regularizers, guarantees Pareto frontier framework. Additionally, simulations real experiments show EIB has ability achieve better domain adaptation results DIB, validates correctness our theories.